Exploring EVENTS¶
Experiments¶
- Visualising Events Dataframe
- Exploring Tags Events
- Calculating Events Description Similarity
- Calculating Events Description Topic Modelling
- Exploring the Schedules of Events
- 5.1 Getting the Frequency of Starting Dates of Events Schedules
- 5.2 Getting the Frequency of End Dates of Events Schedules
- Exploring the Schedules of Events
- Exploring the Performances Tickets of Events Schedules
- 6.1 Getting the Frequency of Price Tickets
- 6.2 Getting the frequency of type (Standard, Children) tickets
- 6.3 Exploring Performances Places - ATENTION: Merging information with "places" dataframe!
- 6.3.1 Frequency of Performances per town
- 6.3.2 Frequency of Type tickets per town
- 6.3.3 Frequency of Price tickets type per town
- 6.3.4 Frequency of Max_Price tickets per town
- 6.3.4.1 Frequency of Free tickets per town
- 6.3.4.2 Frequency of No Free tickets per town
- 6.4 Selecting Scottish Cities: Edinburgh, Glasgow, Dundee, Perth, Inverness, Aberdeen, St Andrews
- 6.4.1 Frequency of Price Tickets per Scottish City
- 6.4.2 Frequency of Type Tickets per Scottish City
- 6.4.3 Frequency of Schedules Dates per Event and per Scottish City
- 6.4.4.Grouping Schedules per Event and Scottish City
- 6.4.5 Exploring Tags per Schedule and Scottish Cities
- 6.4.5.1 Exploring the Frequency of schedules tags for Edinburgh
- 6.4.5.2 Exploring the Frequency of schedules tags for Glasgow
- 6.4.6 Histograms of starting/end schedules dates for Edinburgh
- 6.4.7 Working with Schedule tags, Scottish cities, Starting/End Time
- 6.4.7.1 Frequency of schedules Starting Date in Scottish City
- 6.4.7.2 Frequency of schedules Ending Date in Scottish City
- 6.4.7.3 Scheduled tags and Starting Dates in Scottish City
- 6.4.7.4 Scheduled tags and Starting Dates in Scottish City
- Exploring the Performances Tickets of Events Schedules
0. Importing libraries and loading the json file with 5000 events to a dataframe¶
In [135]:
import json
import pandas as pd
import plotly.express as px
import os
from bertopic import BERTopic
from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity
import pickle
import plotly.graph_objects as go
import numpy as np
from gensim.parsing.preprocessing import remove_stopwords
import re
In [2]:
with open('dataset/sample_20190501.json', 'r') as f:
data = json.load(f)
print(len(data["events"]))
events=data["events"]
df = pd.DataFrame(events)
1. Visualizing the events dataframe¶
In [3]:
df
Out[3]:
In [4]:
## selecting some columns
Experiment 2: Exploring Tags Events¶
We are going to separete the elements stored in each tag list into new rows.
In [5]:
df["tags"][0:5]
Out[5]:
In [6]:
df_tags=df.explode('tags')
In [7]:
df_tags
Out[7]:
In [8]:
g_tags=df_tags.groupby(['tags']).size().reset_index()
g_tags=g_tags.rename(columns={0: "number_of_times"}).sort_values(by=['number_of_times'], ascending=False)
g_tags
Out[8]:
In [9]:
fig = px.line(g_tags, x="tags", y="number_of_times", title='Number of times that each tag appears')
fig.show()
Experiment 3: Description Similarity¶
Exploding the column description¶
Given a description cell, with a list of descriptions, we will create new row per element in that list.
In [10]:
df["descriptions"][0:5]
Out[10]:
In [11]:
df_descriptions=df.explode('descriptions')
In [12]:
df_d=pd.concat([df_descriptions.drop(['descriptions'], axis=1), df_descriptions['descriptions'].apply(pd.Series)], axis=1)
In [13]:
df_desc=df_d[["event_id", "description"]]
In [14]:
df_desc
Out[14]:
Finding similar descriptions events - Deep Learning - Transformers¶
In [15]:
# remving the rows which description is empty
df_desc1=df_desc.dropna(subset=['description']).reset_index()
In [16]:
df_desc1[0:5]
Out[16]:
In [17]:
# total number of rows with descriptions
df_desc1.shape[0]
Out[17]:
In [18]:
#selecting the description colum
documents=df_desc1["description"].values
In [20]:
#d=documents[0:100]
In [136]:
def clean_documents(text):
text = re.sub(r'\S*@\S*\s?', '', text, flags=re.MULTILINE) # remove email
text = re.sub(r'http\S+', '', text, flags=re.MULTILINE) # remove web addresses
text = re.sub("\'", "", text) # remove single quotes
text = remove_stopwords(text)
return text
Going to store cleanned documents in d¶
In [137]:
d=[]
for text in documents:
d.append(clean_documents(text))
In [138]:
# Using all-MiniLM-L6-v2 Transformer
model = SentenceTransformer('all-MiniLM-L6-v2')
In [139]:
#Training our text_embeddings - using the descriptions available & all-MiniLM-L6-v2 Transformer
text_embeddings = model.encode(d, batch_size = 8, show_progress_bar = True)
In [140]:
np.shape(text_embeddings)
Out[140]:
In [141]:
### A small example how to get an embedding vector from a description
In [142]:
first_description=df_desc1["description"].iloc[0]
first_description
first_description_embedding= model.encode(first_description, batch_size = 8, show_progress_bar = True)
Finding the similarity between documents¶
In [143]:
similarity_def=cosine_similarity(
[first_description_embedding],
text_embeddings)
In [144]:
similarities = cosine_similarity(text_embeddings)
print('pairwise dense output:\n {}\n'.format(similarities))
In [145]:
similarities_sorted = similarities.argsort()
similarities_sorted
Out[145]:
In [146]:
id_1 = []
id_2 = []
score = []
for index,array in enumerate(similarities_sorted):
p=len(array)
id_1.append(index)
id_2.append(array[-2])
score.append(similarities[index][array[-2]])
index_df = pd.DataFrame({'id_1' : id_1,
'id_2' : id_2,
'score' : score})
print(p)
In [147]:
index_df
Out[147]:
Finding the first 10 similar definitions given the document 3¶
In [148]:
## Lets take the document 3
doc_index =3
documents[3]
Out[148]:
In [149]:
results={}
for i in range(-2, -12, -1):
similar_index=similarities_sorted[doc_index][i]
rank=similarities[doc_index][similar_index]
results[similar_index]=[rank]
In [150]:
results
Out[150]:
Experiment 4: Description Topic Modelling - Deep Learning - BERTopic¶
Lets find the topic modelling of our descriptions We are going to use the text_embeddings calculated in the previous phase.
In [151]:
len(documents)
Out[151]:
Atention - using the "cleanned" documets -- d¶
In [152]:
topic_model = BERTopic(min_topic_size=20).fit(d, text_embeddings)
In [169]:
topics, probs = topic_model.transform(d, text_embeddings)
Visualizing our topics¶
In [170]:
topic_model.visualize_topics()
In [171]:
#### Visualzing the first 5 keywords of our first 5 topics
In [172]:
topic_model.visualize_barchart()
Visualizing the similarity between topics¶
In [173]:
topic_model.visualize_heatmap()
Getting the frequency of each topic.¶
We should always ignore the first -1 topic.
In [174]:
#Lets see the frequency of the first 10 topics
topic_model.get_topic_freq()[0:10]
Out[174]:
In [175]:
print("Number of topics found %s" %len(topic_model.get_topic_freq()))
Visualizing the keywords of our topics.¶
In [176]:
#topic_model.get_topics()
In [177]:
document_6_topic=topics[6]
print("The topic of the document 6 is %s " %document_6_topic)
In [178]:
topic_model.get_topic(0)
Out[178]:
In [179]:
df_desc1["description"].iloc[6]
Out[179]:
Experiment 5: Exploring the Schedules of Events¶
- 1 Event can have 1 to N Schedules.
- 1 Schedule is in 1 Place
- 1 Schedule can have 1 to N Performances
- 1 Peformance can have 1 to N Tickets
- 1 Ticket has a max_price, min_price, currency.
Lets starting exploding the schedules column
In [50]:
df["schedules"]
Out[50]:
In [51]:
df_schedules=df
df_schedules.rename(columns={'tags':'event_tags'}, inplace=True)
df_schedules.rename(columns={'name':'event_name'}, inplace=True)
df_schedules.rename(columns={'links':'event_links'}, inplace=True)
df_schedules=df.explode('schedules')
#df_schedules
df_s=pd.concat([df_schedules.drop(['schedules'], axis=1), df_schedules['schedules'].apply(pd.Series)], axis=1)
In [52]:
df_s.iloc[0]
Out[52]:
Getting the Frequency of Starting Dates of Events Schedules¶
In [53]:
df_start=df_s.groupby([pd.to_datetime(df_s['start_ts'])]).size().reset_index()
df_start=df_start.rename(columns={0: "number_of_times"})
df_start=df_start.sort_values(by=['number_of_times'], ascending=False)
df_start.reset_index()
Out[53]:
Visualizing the previous Start_Ts Schedules Events Freq.¶
In [55]:
fig = px.histogram(df_start, x='start_ts', y="number_of_times", title="Frequency of Starts Dates Schedules")
fig.show()
Getting the Frequency of End Dates of Events Schedules¶
In [56]:
df_end=df_s.groupby([pd.to_datetime(df_s['end_ts'])]).size().reset_index()
df_end=df_end.rename(columns={0: "number_of_times"})
df_end=df_end.sort_values(by=['number_of_times'], ascending=False)
df_end.reset_index()
fig = px.histogram(df_end, x='end_ts', y="number_of_times", title="Frequency of End Dates Schedules")
fig.show()
Experiment 6: Exploring the Performances Tickets of Events Schedules¶
- 1 Event can have 1 to N Schedules.
- 1 Schedule is in 1 Place
- 1 Schedule can have 1 to N Performances
- 1 Peformance can have 1 to N Tickets
- 1 Ticket has a max_price, min_price, currency.
Lets starting exploding the performance column. We can not explode the performance column, if we hadnt have exploded the schedules column before. For that reason, we are using df_s dataframe, which has already exploded the schedules column.
In [57]:
df_s
Out[57]:
In [58]:
a=df_s[["event_id", "event_name", "performances", "event_tags", "start_ts", "end_ts", "place_id"]]
df_p=a.explode("performances")
In [59]:
df_p
Out[59]:
In [60]:
df_p=pd.concat([df_p.drop(['performances'], axis=1), df_p['performances'].apply(pd.Series)], axis=1)
In [61]:
df_p[0:2]
Out[61]:
Exploring tickets¶
Now we have to explode the tickets column. We are going to remove the rows which tickets information is empty.
In [62]:
df_p=df_p.dropna(subset=['tickets'])
Since we dont need all the columns, we have selects a few of them.
In [63]:
df_t=df_p[["event_id", "event_name", "descriptions", "event_tags", "tickets", "place_id", "start_ts", "end_ts"]]
In [64]:
df_t[0:5]
Out[64]:
In [65]:
df_t1=df_t.explode("tickets")
Now we are going to transform the max, and min prices of tickets to numeric values.
In [66]:
df_tickets=pd.concat([df_t1.drop(['tickets'], axis=1), df_t1['tickets'].apply(pd.Series)], axis=1)
df_tickets['min_price'] = pd.to_numeric(df_tickets['min_price'])
df_tickets['max_price'] = pd.to_numeric(df_tickets['max_price'])
df_tickets['min_price']= df_tickets['min_price'].fillna(0)
df_tickets['max_price']= df_tickets['max_price'].fillna(0)
In [67]:
df_tickets[0:5]
Out[67]:
Experiment 6.1: Getting the Frequency of Price Tickets¶
We are working just with max_price.
In [68]:
g_maxp=df_tickets.groupby(['max_price']).size().reset_index()
g_maxp=g_maxp.rename(columns={0: "number_of_times"})
#g_maxp=g_maxp.sort_values(by=['number_of_times'], ascending=False)
free_tickets=g_maxp[0:1]
## Removing FREE TICKETS
g_maxp=g_maxp.drop([0])
###
g_maxp[:]
Out[68]:
In [69]:
fig = px.line(g_maxp, x="max_price", y="number_of_times", title='Frequency of price tickets')
fig.show()
In [70]:
print("The number of free tickets is: %s" %free_tickets["number_of_times"].values[0])
Experiment 6.2: Getting the frequency of type (Standard, Children) tickets¶
In [133]:
tickets_type=df_tickets.groupby(['type']).size().reset_index()
tickets_type=tickets_type.rename(columns={0: "number_of_times"}).sort_values(by=['number_of_times'], ascending=False)
tickets_type
Out[133]:
In [134]:
px.histogram(tickets_type, x="type", y="number_of_times", histfunc="sum", color="type", title='Frequency of type tickets')
6.3 Exploring Performances Places¶
In [132]:
df_tickets["place_id"]
Out[132]:
Creating places dataframe¶
In [74]:
data="dataset/sample_20180501.json"
with open('dataset/sample_20180501.json', 'r') as f:
data = json.load(f)
print(len(data["places"]))
places=data["places"]
df_places = pd.DataFrame(places)
In [75]:
df_place = df_tickets.merge(df_places, on=['place_id','place_id'])
In [76]:
df_place.shape[0]
Out[76]:
6.3.1 Frequency of Performances per Town¶
In [77]:
df_town=df_place.dropna(subset=['town'])
town=df_town.groupby(['town']).size().reset_index()
town=town.rename(columns={0: "number_of_times"})
town=town.drop([0])
In [78]:
town=town.sort_values(by=['number_of_times'], ascending=False)
town
Out[78]:
In [79]:
px.scatter(town, x="town",y='number_of_times', color='number_of_times', size="number_of_times", size_max=60, title="Frequency of Performances per Town")
6.3.2 Frequency of Type tickets per town¶
In [80]:
town_type=df_town.groupby(['town', 'type']).size().reset_index()
town_type=town_type.rename(columns={0: "number_of_times"})
town_type=town_type[town_type["town"]!=""]
In [81]:
town_type=town_type.sort_values(by=['number_of_times'], ascending=False)
town_type
Out[81]:
In [82]:
fig = px.scatter(town_type, x='town', y='type', color='number_of_times', title="Frequency of type tickets per town")
fig.show()
In [83]:
px.scatter(town_type, x="town",y='type', color='number_of_times', size="number_of_times", size_max=60, title="Frequency of performances type tickets per town")
6.3.3. Frequency of Max_Price tickets per towns¶
In [84]:
a=df_town[["town", "max_price"]]
a=a[a["town"]!=""]
town_price=a.groupby(['town', 'max_price']).size().reset_index()
town_price=town_price.rename(columns={0: "number_of_times"})
town_price=town_price.sort_values(by=['number_of_times'], ascending=False)
town_price
Out[84]:
6.3.3.1. Frequency of free tickets per town¶
In [85]:
free_town_price=town_price[town_price["max_price"]== 0.0]
free_town_price
Out[85]:
In [86]:
fig = px.bar(free_town_price, x='town', y='number_of_times', color='number_of_times', barmode='group', title="Frequency of Free Tickets per Town")
fig.show()
6.3.3.1. Frequency of No free tickets per town¶
In [87]:
town_price=town_price[town_price["max_price"]!= 0.0]
town_price
Out[87]:
In [88]:
fig = px.bar(town_price, x='town', y='max_price', color='number_of_times', barmode='group', title="Frequency of Price Tickets per Town")
fig.show()
In [89]:
town_price.groupby(["town"]).sum().sort_values(by=['max_price'], ascending=False)
Out[89]:
6.4 Selecting Scottish Cities: Edinburgh, Glasgow, Dundee, Perth, Inverness, Aberdeen, St Andrews¶
6.4.1 Frequency of Price Tickets per Scottish City¶
In [90]:
scot_towns_price=town_price[town_price['town'].isin(["Edinburgh", "Glasgow", "Perth", "Inverness", "Dundee", "St Andrews", "Aberdeen"])]
In [91]:
scot_towns_price[0:10]
Out[91]:
In [92]:
fig = px.bar(scot_towns_price, x='town', y='max_price', color='number_of_times', barmode='group', title="Frequency of Price Tickets per Scottish City")
fig.show()
In [93]:
scot_towns_price.groupby(["town"]).sum().sort_values(by=['max_price'], ascending=False)
Out[93]:
6.4.2 Frequency of Type Tickets per Scottish City¶
In [94]:
scot_towns_type=town_type[town_type['town'].isin(["Edinburgh", "Glasgow", "Perth", "Inverness", "Dundee", "St Andrews", "Aberdeen"])]
In [95]:
scot_towns_type[0:10]
Out[95]:
In [96]:
fig = px.bar(scot_towns_type, x='town', y='number_of_times', color='type', barmode='group', title="Frequency of Type Tickets per Scottish City")
fig.show()
In [97]:
scot_towns_type.groupby(["town"]).sum()
Out[97]:
In [98]:
df_place.loc[0]
Out[98]:
6.4.3.3 Frequency of Schedules Dates per Event and per Scottish City¶
In [99]:
df_place2=df_place.dropna(subset=['town'])
df_place2
df_scott=df_place2[df_place2['town'].isin(["Edinburgh", "Glasgow", "Perth", "Inverness", "Dundee", "St Andrews", "Aberdeen"])]
df_scott=df_scott[["event_id", "event_name", "event_tags", "town", "start_ts", "end_ts"]]
df_scott[0:3]
Out[99]:
Note: An event can have several schedules. And a schedule has an starting and end date. Therefore, an event can have several starting and end dates.
In [100]:
fig = px.scatter(df_scott, x='start_ts', y="event_name", title="Frequency of starting date per event in Scottish cities")
fig.show()
In [101]:
fig = px.scatter(df_scott, x='end_ts', y="event_name", title="Frequency of ending date per event in Scottish cities")
fig.show()
6.4.4 Grouping Schedules per Event and Scottish City¶
In [102]:
scott_schedule=df_scott.groupby(['event_name', 'town']).size().reset_index()
scott_schedule=scott_schedule.rename(columns={0: "number_of_times"})
scott_schedule=scott_schedule.sort_values(by=['number_of_times'], ascending=False)
scott_schedule
Out[102]:
In [103]:
t=scott_schedule.groupby(["event_name"]).sum().sort_values(by=['number_of_times'], ascending=False)
t
Out[103]:
In [104]:
fig = px.bar(t, title="Frequency of Schedules per event")
fig.show()
6.4.5 Exploring Tags per Schedule and Scottish Cities.¶
In [105]:
a=df_scott.reset_index(drop=True)
tags_town=a[["event_tags", "town"]]
tags_town=tags_town.explode("event_tags")
tags_town
Out[105]:
In [106]:
scott_tag=tags_town.groupby(['town', 'event_tags']).size().reset_index()
scott_tag=scott_tag.rename(columns={0: "number_of_times"})
scott_tag=scott_tag.sort_values(by=['number_of_times'], ascending=False)
scott_tag
Out[106]:
In [130]:
fig=px.histogram(scott_tag, x="town", y="number_of_times", histfunc="sum", color="event_tags", title='Frequency of tags in Scottish Cities')
fig.update_layout(legend_traceorder="reversed")
fig.show()
In [108]:
t=scott_tag.groupby(["event_tags"]).sum().sort_values(by=['number_of_times'], ascending=False)
t
Out[108]:
6.4.5.1 Exploring the Frequency of schedules tags for Edinburgh¶
In [109]:
edi_scott_tag=scott_tag[scott_tag['town'].isin(["Edinburgh"])]
edi_scott_tag
Out[109]:
In [110]:
edi_scott_tag.groupby(["event_tags"]).sum().sort_values(by=['number_of_times'], ascending=False)
Out[110]:
In [111]:
fig = px.bar(edi_scott_tag, x='town', y='number_of_times', color='event_tags', barmode='group', title="Frequency of schedules tags for Edinburgh")
fig.show()
6.4.6 Histograms of starting/end schedules dates for Edinburgh¶
In [112]:
scott_start=df_scott.groupby([pd.to_datetime(df_scott['start_ts']), "town"]).size().reset_index()
scott_start=scott_start.rename(columns={0: "number_of_times"})
scott_start=scott_start.sort_values(by=['number_of_times'], ascending=False)
scott_start.reset_index()
Out[112]:
In [113]:
ed_scott_start=scott_start[scott_start['town'].isin(["Edinburgh"])].reset_index()
ed_scott_start.groupby(["start_ts"]).sum().sort_values(by=['number_of_times'], ascending=False)
#fig = px.bar(ed_scott_start, x='town', y='number_of_times', color='start_ts', barmode='group', title="Frequency of starting date schedules for Edinburgh")
#fig.show()
Out[113]:
In [114]:
scott_end=df_scott.groupby([pd.to_datetime(df_scott['end_ts']), "town"]).size().reset_index()
scott_end=scott_end.rename(columns={0: "number_of_times"})
scott_end=scott_end.sort_values(by=['number_of_times'], ascending=False)
scott_end.reset_index()
ed_scott_end=scott_end[scott_end['town'].isin(["Edinburgh"])].reset_index()
ed_scott_end.groupby(["end_ts"]).sum().sort_values(by=['number_of_times'], ascending=False)
#fig = px.bar(ed_scott_end, x='town', y='number_of_times', color='end_ts', barmode='group', title="Frequency of ending date schedules for Edinburgh")
#fig.show()
Out[114]:
In [115]:
fig = px.histogram(ed_scott_start, x='start_ts', y="number_of_times", title="Histogram of Schedules Starting Dates for Edinburgh")
fig.show()
In [116]:
fig = px.histogram(scott_start, x='start_ts', y="number_of_times", title="Histogram of Schedules Starting Dates for Scottish Cities")
fig.show()
In [117]:
fig = px.histogram(scott_end, x='end_ts', y="number_of_times", title="Histogram of Schedules Ending Dates for Scottish Cities")
fig.show()
In [118]:
fig = px.histogram(scott_end, x="end_ts", y="number_of_times", histfunc="sum", title="Histogram on Date Axes")
fig.update_traces(xbins_size="M1")
fig.update_xaxes(showgrid=True, ticklabelmode="period", dtick="M1", tickformat="%b\n%Y")
fig.update_layout(bargap=0.1)
fig.add_trace(go.Scatter(mode="markers", x=scott_end["end_ts"], y=scott_end["number_of_times"], name="daily"))
fig.show()
6.4.7 Working with Schedule tags, Scottish cities, Starting/End Time¶
In [119]:
b=df_scott.reset_index(drop=True)
tag_town_time=b[["event_tags", "town", "start_ts", "end_ts"]]
tag_town_time=tag_town_time.explode("event_tags")
tag_town_time
Out[119]:
In [120]:
scott_tag_end=tag_town_time.groupby([pd.to_datetime(tag_town_time['end_ts']), "event_tags"]).size().reset_index()
scott_tag_end=scott_tag_end.rename(columns={0: "number_of_times"})
scott_tag_end=scott_tag_end.sort_values(by=['number_of_times'], ascending=False)
scott_tag_start=tag_town_time.groupby([pd.to_datetime(tag_town_time['start_ts']), "event_tags"]).size().reset_index()
scott_tag_start=scott_tag_start.rename(columns={0: "number_of_times"})
scott_tag_start=scott_tag_start.sort_values(by=['number_of_times'], ascending=False)
In [121]:
scott_tag_start
Out[121]:
6.4.7.1 Frequency of schedules Starting Date in Scottish City¶
In [122]:
#fig = px.bar(scott_tag_start, x='event_tags', y='start_ts', color='number_of_times', barmode='group', title="Frequency of schedules tags per Scottish City")
#fig.show()
fig = px.scatter(scott_tag_start, x='start_ts', y='number_of_times', title="Frequency of schedules Starting Date in Scottish City.")
fig.show()
6.4.7.2 Frequency of schedules Ending Date in Scottish City¶
In [123]:
fig = px.scatter(scott_tag_end, x='end_ts', y='number_of_times', title="Frequency of schedules Ending Date in Scottish City.")
fig.show()
6.4.7.3 Scheduled tags and Starting Dates in Scottish City¶
In [124]:
fig = px.scatter(scott_tag_start, x='start_ts', y='event_tags', title="Scheduled Tags and Starting Dates in Scottish City.")
fig.show()
6.4.7.3 Scheduled Tags and Ending Dates in Scottish City¶
In [125]:
fig = px.scatter(scott_tag_end, x='end_ts', y='event_tags', title="Scheduled Tags and Ending Dates in Scottish City.")
fig.show()